SDR replaces VectorHelpers #320

breznak · 2019-03-07T18:20:30Z

I want to replace some/most functionality of VectorHelpers by SDR.

SP uses SDR
Hotgym uses SDR
make SDR_dense_t templated
remove VectorHelpers::sparseToBinary, binaryToSparse

char is more space efficient, but our algorithms work with UInt for indices, so to avoid needed conversion. Also, internally mostly sdr_sparse_t is used for SDR data representation, so this should not have a significant impact on SDR

instead of VectorHelpers

as those are useful in algorithms working with SDRs

breznak

RFC on how to change SDR_dense_t? hard-code vector, or template?

src/examples/hotgym/HelloSPTP.cpp

breznak · 2019-03-07T18:24:40Z

src/examples/hotgym/HelloSPTP.cpp

@@ -171,11 +170,11 @@ Real64 BenchmarkHotgym::run(UInt EPOCHS, bool useSPlocal, bool useSPglobal, bool

    //Anomaly (pure x likelihood)
    tAn.start();
-    res = an.compute(outSP /*active*/, prevPred_ /*prev predicted*/);
+    res = an.compute(outSP.getDense() /*active*/, prevPred_ /*prev predicted*/);


I need SDR_dense_t to be vector UInt, not current vector Byte.
Should I set it just there.
Or make it templated?
It seems to be a big change which breaks a lot of stuff in SDR.

breznak · 2019-03-07T18:25:37Z

src/nupic/types/Sdr.hpp

@@ -35,7 +35,7 @@ using namespace std;

 namespace nupic {

-typedef vector<Byte>          SDR_dense_t;
+typedef vector<UInt>          SDR_dense_t; //TODO add templated types for SDR<DenseElemT, SparseElemT, CoordElemT> + default SDR<Byte, UInt, UInt> 


this is the breaking change. I think it'd be good to have this option.

Are you saying that the SDR internal dense format should be UInt? That is not a good idea.

just a convenience wrapper

src/nupic/types/Sdr.cpp

src/examples/hotgym/HelloSPTP.cpp

This reverts commit f651b42.

so SDR a = some_sdr; is now allowed. This is an ugly commit, too many changes. It mixes SDR moved under ns nupic::sdr; and some namespace fixes along the way. And need to replace &size, &dimensions with getter methods size(), dimensions() to allow the copy constructor which is needed for likes of: SDR a = sp.someFunctionReturningSDR();

breznak

not happy with the last, large commit, but I'd like to pull it through.

ns nupic::sdr::SDR;
SDR a = getsomeSDR(); //allows copy constructor

src/nupic/types/Sdr.hpp

ctrl-z-9000-times · 2019-03-08T13:58:39Z

The SDR already allows the copy constructor...
This compiles:

SDR A({ 1000 });
SDR B = A;

This reverts commit d4cf24d.

this allows to avoid deepcopy and no need for copy-constructor for SDRs

ctrl-z-9000-times

Overall this looks good.
Some questions about the changes to the SDR testcases though.

src/nupic/algorithms/SpatialPooler.cpp

ctrl-z-9000-times · 2019-05-06T12:43:20Z

src/test/unit/types/SdrTest.cpp

    ASSERT_NE( before, after ); // not a copy.
    ASSERT_EQ( after, data ); // correct data buffer.
 }

-TEST(SdrTest, TestSetDenseByte) {


Where did this test case go?

I thought it was not needed due to some casts, but I'll revisit it again!
Btw, do we need all 3 variants for SDR::setXXX() ? Can we do just with SDR.setDense(SDR_dense_t)?

The overloads are:

void setDense( SDR_dense_t &value ); void setDense( const std::vector<T> &value ) { void setDense( const T *value ) {

constant & non-constant vectors need different code.
If vector of wrong data type it will do the converstion to Byte.
Raw pointer is for legacy code.

So yes, I think we need all of these.

src/test/unit/types/SdrTest.cpp

ctrl-z-9000-times

This looks better.

There are still a two issues with the SDR unit tests, where you've changed a data type. The SDR has a big matrix of different data types which should work, even though they are not the exact same type which the SDR uses internally.

Also, I don't see any changes to VectorHelpers. Isn't the purpose of this PR to remove it?

ctrl-z-9000-times · 2019-06-13T15:08:37Z

src/test/unit/types/SdrTest.cpp

    ASSERT_NE( before, after ); // not a copy.
    ASSERT_EQ( after, data ); // correct data buffer.
 }

+TEST(SdrTest, TestSetDenseUInt) {
+    SDR a({11, 10, 4});
+    auto vec = SDR_dense_t(a.size, 1);


This data type should not have changed. The test name explicitly calls for UInt not Byte

The SDR has a big matrix of different data types which should work, even though they are not the exact same type which the SDR uses internally.

yes, I made a misunderstood these. will need to get all of those back.

Also, I don't see any changes to VectorHelpers. Isn't the purpose of this PR to remove it?

yes, I've just updated the old PR to see how it fares. VectorH should be removed in this PR.

Also, I'm getting failing "exact output" tests but I don't see anything that should cause that (at the first glance)

ctrl-z-9000-times · 2019-06-13T15:45:41Z

@breznak,
I fixed the SDR unit tests like I want them, and I added a few comments to hopefully reduce confusions in the future.

breznak · 2019-06-13T16:11:58Z

fixed the SDR unit tests like I want them, and I added a few comments

thanks for fixing those!

ctrl-z-9000-times · 2019-06-13T16:25:45Z

src/nupic/algorithms/SpatialPooler.cpp

@@ -548,8 +550,7 @@ vector<UInt> SpatialPooler::initMapPotential_(UInt column, bool wrapAround) {

  const UInt numPotential = (UInt)round(columnInputs.size() * potentialPct_);
  const auto selectedInputs = rng_.sample<UInt>(columnInputs, numPotential);
-  const vector<UInt> potential = VectorHelpers::sparseToBinary<UInt>(selectedInputs, numInputs_);
-  return potential;
+  potential.setSparse(selectedInputs);


This is not sorted! This is the bug causing the exactOutput test to fail.

Because the potential pool is not sorted, the initial Permanences are randomly generated differently than before.

This also fails in debug mode where there is an ASSERT( is_sorted ) for this type of error.

most of the functionality now provided by SDR.hpp class

breznak added 8 commits March 7, 2019 15:20

SDR: make sdr_dense_t element UInt, from char

e2fa54e

char is more space efficient, but our algorithms work with UInt for indices, so to avoid needed conversion. Also, internally mostly sdr_sparse_t is used for SDR data representation, so this should not have a significant impact on SDR

SpatialPooler: use SDR for conversion from sparse/dense

70cdb1a

instead of VectorHelpers

SDR: make set*Inplace() public

f651b42

as those are useful in algorithms working with SDRs

SDRProxy fix UInt in dense

858d1f3

Hotgym: use SDR instead of VectorHelpers

3391f78

SDR: equals uses getSparse internally

4d793ab

fix pybindings for SDR getDense change to vector<UInt>

c2bcb04

SDR: update unit test

6d68136

breznak added code code enhancement, optimization, cleanup..programmer stuff sdr labels Mar 7, 2019

breznak commented Mar 7, 2019

View reviewed changes

breznak added 4 commits March 7, 2019 20:30

Hotgym: use compute(SDR) calls

bdb7ffd

SDR: TMP rever to Byte for dense()

83d0e9c

TM: add compute(SDR, learn) without extra inputs wrapper

9cfd73d

just a convenience wrapper

SpatialPooler: use SDR, sparse initMapPotentials

6371aa2

breznak self-assigned this Mar 7, 2019

ctrl-z-9000-times reviewed Mar 7, 2019

View reviewed changes

src/nupic/types/Sdr.cpp Outdated Show resolved Hide resolved

src/examples/hotgym/HelloSPTP.cpp Outdated Show resolved Hide resolved

breznak added 4 commits March 7, 2019 23:54

Revert "SDR: make set*Inplace() public"

eb016bc

This reverts commit f651b42.

Hotgym fix SDR use

1f430b2

Hotgym code fix

11a6190

breznak commented Mar 8, 2019

View reviewed changes

src/nupic/types/Sdr.hpp Outdated Show resolved Hide resolved

src/nupic/types/Sdr.hpp Outdated Show resolved Hide resolved

ctrl-z-9000-times reviewed Mar 8, 2019

View reviewed changes

src/nupic/types/Sdr.hpp Outdated Show resolved Hide resolved

breznak added 6 commits March 12, 2019 11:55

Revert "SDR: move to namespace nupic::sdr, allow copy-constructor"

dc7fb3c

This reverts commit d4cf24d.

Merge branch 'master_community' into sdr_cleanup

436c600

SP: initMapPotential takes output SDR as its parameter

96302f9

this allows to avoid deepcopy and no need for copy-constructor for SDRs

fix py binding

e303d36

SP: fix test for initMapPotential

a76cd0e

SDR: again use dense for equals

53dd7ac

breznak added 8 commits April 3, 2019 10:10

Merge branch 'master_community' into sdr_cleanup

9a9575d

merge fixes

f2d1a0d

Merge branch 'master_community' into sdr_cleanup

2b1cf93

Merge branch 'master_community' into sdr_cleanup

2823d4e

Connections: adaptSegment small cleanup

3ca10e4

Merge branch 'master_community' into sdr_cleanup

a9912f3

Merge branch 'master_community' into sdr_cleanup

ffeb5a6

post merge conflicts

d37daf9

breznak added the ready label May 6, 2019

breznak requested review from dkeeney and ctrl-z-9000-times May 6, 2019 00:33

breznak added 2 commits May 6, 2019 08:28

remove whitespace changes

954013e

Merge branch 'master_community' into sdr_cleanup

4a850b9

breznak added in_progress and removed ready labels May 6, 2019

ctrl-z-9000-times requested changes May 6, 2019

View reviewed changes

breznak added 3 commits May 6, 2019 21:06

Merge branch 'master_community' into sdr_cleanup

0243edc

Merge branch 'master_community' into sdr_cleanup

a324d30

Merge branch 'master_community' into sdr_cleanup

a738702

ctrl-z-9000-times mentioned this pull request Jun 12, 2019

Find unused methods, bitrot #175

Open

breznak added 3 commits June 13, 2019 16:01

Merge branch 'master_community' into sdr_cleanup

7643e34

fix merge conflicts

cb1ce21

some review feedback

3de7990

ctrl-z-9000-times requested changes Jun 13, 2019

View reviewed changes

SDR Unit Tests: fixes & comments

409c9d4

ctrl-z-9000-times reviewed Jun 13, 2019

View reviewed changes

breznak added 2 commits June 13, 2019 19:29

fix sorted inputs for SDR

2604dd0

remove obsoleted VectorHelpers file

a1d9a14

most of the functionality now provided by SDR.hpp class

SDR replaces VectorHelpers #320

Are you sure you want to change the base?

SDR replaces VectorHelpers #320

Uh oh!

Conversation

breznak commented Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

breznak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

breznak Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

breznak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ctrl-z-9000-times commented Mar 8, 2019

Uh oh!

ctrl-z-9000-times left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ctrl-z-9000-times left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ctrl-z-9000-times commented Jun 13, 2019

Uh oh!

breznak commented Jun 13, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

breznak commented Mar 7, 2019 •

edited

Loading

breznak Mar 7, 2019 •

edited

Loading